1. Field of the Invention
This invention relates to memory subsystems and, more particularly, to the topology of a memory system architecture.
2. Description of the Related Art
Over the years, demands placed upon computing systems have increased. As demands increase and technology has advanced, efforts have been made to both increase the capacity of memory systems and reduce memory system latencies. In recent years, proposals have been made for fully-buffered dual-inline memory modules (FB-DIMMs). Traditionally, DIMMs have been configured to buffer both addresses and commands, but the data interface itself has been unbuffered. For a variety of reasons, including loading considerations, such designs have been unable to provide sufficiently high data rates when multiple DIMMs are placed on the same data channel. As a result, DIMMs that buffer data (i.e., FB-DIMMS), as well as addresses and commands, have been proposed. One such proposal includes a buffer chip on each DIMM and uses high-speed unidirectional point-to-point signaling between the memory controller and between DIMMs.
FIG. 1 (prior art) illustrates one embodiment of such a proposal which is being authored by the Joint Electron Device Engineering Council (JEDEC). Among other things, the proposal concerns FB-DIMMs which offer greater capacity, including scalability of up to 192 GB, and as many as six channels and eight DIMMs per channel. In the example shown, a system 100 is depicted which includes a processing unit 110, a host 120, and memory modules 130A, 130B, 130C, 130D, 130E, 130F. Each of the memory modules 130 are FB-DIMMs 130 and are configured on a channel 156 from the host 120 (e.g., a memory controller) which is in turn coupled to a processing unit 110. Elements referred to herein with a particular reference number followed by a letter will be collectively referred to by the reference number alone. For example, memory modules 130A-130F may be collectively referred to as memory modules 130. In addition, each of the FB-DIMMs 130A-130F includes an advanced memory buffer (AMB) 150A-150F, respectively. Each AMB 150 on the channel 156 must be uniquely identifiable to be addressable by the system.
A first FB-DIMM 130A is coupled to the host 120 by two separate links (160, 170). The first of these links 160 may be referred to as a “downstream” or “southbound” link, in which the first FB-DIMM 130A receives data and/or commands from host 120. The second of these links 170 may be referred to as an “upstream” or “northbound” link, by which data is conveyed from FB-DIMMs 130 to host 120. The remainder of the FB-DIMMs 130 in the embodiment shown are coupled to each other through a plurality of upstream and downstream links as illustrated. In general, a link may be referred to as an upstream link if information conveyed through the link is flowing towards the host 120, while a link may be referred to as a downstream link if information conveyed through the link is flowing away from host 120. The FB-DIMMs 130 are coupled to each other in what is commonly referred to as a “daisy-chain” arrangement.
Generally speaking, downstream communication takes place by sending fixed-size frames on the downstream links, with each frame being sent to each FB-DIMM 130 in turn. These downstream frames are generally pipelined one after another without any gaps. The content of all downstream frames is conveyed from the host 120. Upstream communication takes place by sending a pipelined sequence of fixed-size frames on the upstream links. The content of upstream frames may be updated by FB-DIMMs 130 as they are forwarded by them. All read and write commands, as well as write data, are sent on the downstream links. All read data is sent on the upstream links.
Scheduling of reads and writes is performed by host/memory controller 120. In order to simplify the scheduling, all read data is returned in the order in which it was requested. Thus, data read from a particular FB-DIMM 130 is placed in the first available upstream frame following the availability of the data, while data read from the first FB-DIMM 130A is delayed until the appropriate upstream frame passes it. Each FB-DIMM 130 knows the required latency from when it receives the downstream read command to when it inserts the corresponding read data into an upstream packet (these latencies must be deterministic; they are proposed to be configured during power-on of the channel, although there are questions about the deterministic nature of this initialization scheme). For example, if the propagation delay from one FB-DIMM to another is D ns and the latency for reading data on an FB-DIMM is R ns, and if there are no delays due to framing alignments, the Nth FB-DIMM from the host 120 would provide its read data R ns after receiving its command and the FB-DIMM nearest to the host would insert its read data into a frame R+2*(N−1)*D ns after receiving its command. In general, regardless of which FB-DIMM 130 is being read, the corresponding read data would arrive at the host 120 R+2*N*D ns after the command was sent out. Example values of N may typically vary from 1 to 8 or 16 and typical values of D may be in the range of 2 ns.
While the above scheme may provide a simple means for scheduling read data, it may also suffer from potentially high latencies when N is large (e.g. 8 or 16). In addition, in order to deal with such latencies, it may also require buffering on each FB-DIMM 20. Further, it has inefficiencies related to scheduling frames for the read commands and the read data.
In view of the above, an effective and efficient memory architecture is desired.